Algorithms and Applications
In this paper we explore some implications of view-ing graphs as geometric objects. This approach of-fers a new perspective on a number of graph-theoretic and algorithmic problems. There are several ways to model graphs geometrically and our main concern here is with geometric representations that respect the met-ric of the (possibly weighted) graph. Given a graph G we map its vertices to a normed space in an attempt to (i) Keep down the dimension of the host space and (ii) Guarantee a small distortion, i.e., make sure that distances between vertices in G closely match the dis-tances between their geometric images. In this paper we develop efficient algorithms for em-bedding graphs low-dimensionally with a small distor-tion. Further algorithmic applications include: 0 A simple, unified approach to a number of prob-lems on multicommodity flows, including the Leighton-Rae Theorem [29] and some of its ex-tensions. 0 For graphs embeddable in low-dimensional spaces with a small distortion, we can find low-diameter decompositions (in the sense of [4] and [34]). The parameters of the decomposition depend only on the dimension and the distortion and not on the size of the graph. 0 In graphs embedded this way, small balanced separators can be found efficiently. Faithful low-dimensional representations of statisti-cal data allow for meaningful and efficient cluster-ing, which is one of the most basic tasks in pattern-recognition. For the (mostly heuristic) methods used
Covering Analysis of the Greedy Algorithm for Partial Cover.

An Analysis of Relevance Vector Machine Regression.

Foundations of Intelligent Systems
The aim of this paper is to introduce the theoretical foundations of an approach for intelligent systems development. Derived from semiotics, a classic discipline in human sciences, the theory developed provides a mathematical framework for the concept of knowledge and for knowledge processing. As a result, a new perspective to study and to develop intelligent systems emerges. A taxonomy of elementary types of knowledge is proposed based on the classification of types of signs in semiotics, followed by a another classification of knowledge from the point of view of application in cognitive systems. In addition, we propose the mathematical definition of objects, objects systems and objects networks, to model mathematically the different types of knowledge described. The symbiosis of such key concepts introduces a computational paradigm to develop and implement intelligent systems, called here computational semiotics. 
A Walk from 2%2DNorm SVM to 1%2DNorm SVM.
The 1-norm SVM performs better than the standard 2-norm regularized SVM on problems domains with many irrelevant features. This paper studies how useful the standard SVM is in approximating the 1-norm SVM problem. To this end, we examine a general method that is based on iteratively re-weighting the features and solving a 2-norm optimization problem. The convergence rate of this method is unknown. Previous work indicates that it might require an excessive number of iterations. We study how well we can do with just a small number of iterations. In theory the convergence rate is fast, except for coordinates of the current solution that are close to zero. Our empirical experiments confirm this. In both real-world and synthetic problems with irrelevant features, already one iteration is often enough to produce accuracy as good as or better than that of the 1-norm SVM. Hence, it seems that in these problems we do not need to converge to the 1-norm SVM solution near zero values. The benefit of this approach is that we can build something resembling the 1-norm regularized solver based on any 2-norm regularized solver. This is quick to implement and the solution inherits the good qualities of the solver such as scalability and stability. For linear SVMs the recent advances in efficient solvers make this approach practical. 
A Walk from 2%2DNorm SVM to 1%2DNorm SVM.
The 1-norm SVM performs better than the standard 2-norm regularized SVM on problems domains with many irrelevant features. This paper studies how useful the standard SVM is in approximating the 1-norm SVM problem. To this end, we examine a general method that is based on iteratively re-weighting the features and solving a 2-norm optimization problem. The convergence rate of this method is unknown. Previous work indicates that it might require an excessive number of iterations. We study how well we can do with just a small number of iterations. In theory the convergence rate is fast, except for coordinates of the current solution that are close to zero. Our empirical experiments confirm this. In both real-world and synthetic problems with irrelevant features, already one iteration is often enough to produce accuracy as good as or better than that of the 1-norm SVM. Hence, it seems that in these problems we do not need to converge to the 1-norm SVM solution near zero values. The benefit of this approach is that we can build something resembling the 1-norm regularized solver based on any 2-norm regularized solver. This is quick to implement and the solution inherits the good qualities of the solver such as scalability and stability. For linear SVMs the recent advances in efficient solvers make this approach practical. 


Ranking the Uniformity of Interval Pairs.

Obtaining Low%2DArity Discretizations from Online Data Streams.

Maintaining Optimal Multi%2Dway Splits for Numerical Attributes in Data Streams.

Reducing Splaying by Taking Advantage of Working Sets.
Abstract. Access requests to keys stored into a data structure often exhibit locality of reference in practice. Such a regularity can be modeled, e.g., by working sets. In this paper we study to what extent can the existence of working sets be taken advantage of in splay trees. In order to reduce the number of costly splay operations we monitor for information on the current working set and its change. We introduce a simple algorithm which attempts to splay only when necessary. Under worst-case analysis the algorithm guarantees an amortized logarithmic bound. In empirical experiments it is 5 % more efficient than randomized splay trees and at most 10 % more efficient than the original splay tree. We also briefly analyze the usefulness of the commonly-used Zipf’s distribution as a general model of locality of reference. 1
Following the Perturbed Leader to Gamble at Multi%2Darmed Bandits.
mail:
The cost of offline binary search tree algorithms and the complexity of the request sequence.
mail:
Following the Perturbed Leader to Gamble at Multi%2Darmed Bandits.
mail:


A Voronoi Diagram Approach to Autonomous Clustering.

Poketree A Dynamically Competitive Data Structure with Good Worst%2DCase Performance.
Abstract. We introduce a new O(lg lg n)-competitive binary search tree data structure called poketree that has the advantage of attaining, under worst-case analysis, O(lg n) cost per operation, including updates. Previous O(lg lg n)-competitive binary search tree data structures have not achieved O(lg n) worst-case cost per operation. A standard data structure such as red-black tree or deterministic skip list can be augmented with the dynamic links of a poketree to make it O(lg lg n)-competitive. Our approach also uses less memory per node than previous competitive data structures supporting updates. 1
Practical Approximation of Optimal Multivariate Discretization.

On Following the Perturbed Leader in the Bandit Setting.
mail:
Approximation Algorithms for Minimizing Empirical Error by Axis%2DParallel Hyperplanes.

On Autonomous k%2DMeans Clustering.

On look%2Dahead and pathology in decision tree learning.

The Difficulty of Reduced Error Pruning of Leveled Branching Programs.
Induction of decision trees is one of the most successful approaches to supervised machine learning. Branching programs are a generalization of decision trees and, by the boosting analysis, exponentially more efficiently learnable than decision trees. In experiments this advantage has not been seen to materialize. Decision trees are easy to simplify using pruning. For branching programs no such algorithms are known. We prove that reduced error pruning of branching programs is infeasible. Finding the optimal pruning of a branching program with respect to a set of pruning examples that is separate from the set of training examples is NP-complete. Therefore, we are forced to consider approximate solutions to this problem. We also prove that finding an approximate solution of arbitrary accuracy is computationally intractable. In particular, the reduced error pruning of branching programs is APX-hard.
Efficient Multisplitting Revisited Optima%2DPreserving Elimination of Partition Candidates.

Selective Rademacher Penalization and Reduced Error Pruning of Decision Trees.

Rademacher Penalization over Decision Tree Prunings.
Rademacher penalization is a modern technique for obtaining  data-dependent bounds on the generalization error of classi  ers. It  would appear to be limited to relatively simple hypothesis classes because  of computational complexity issues. In this paper we, nevertheless,  apply Rademacher penalization to the in practice important hypothesis  class of unrestricted decision trees by considering the prunings of a given  decision tree rather than the tree growing phase. Moreover, we generalize  the error-bounding approach from binary classi  cation to multi-class  situations. Our empirical experiments indicate that the proposed new  bounds clearly outperform earlier bounds for decision tree prunings and  provide non-trivial error estimates on real-world data sets.
On Lookahead Heuristics in Decision Tree Learning.

On Decision Boundaries of Na%EFve Bayes in Continuous Domains.

Reduced Error Pruning of branching programs cannot be approximated to within a logarithmic factor.
In this paper, we prove under a plausible complexity hypothesis that Reduced Error Pruning of branching programs is hard to approximate within log    n,forevery#&gt;0, where n is the number of description variables, a measure of the problem&#039;s complexity. The result holds under the assumption that NP problems do not admit deterministic, slightly superpolynomial time algorithms: NP      |I |) ). This improves on a previous result that only had a small constant inapproximability ratio, and puts a fairly strong constraint on the efficiency of potential approximation algorithms. The result also holds for read-once and -branching programs.
Necessary and Sufficient Pre%2Dprocessing in Numerical Range Discretization.

Machine Learning ECML 2002

Flexible view recognition for indoor navigation based on Gabor filters and support vector machines.

Machine Learning ECML 2002

Principles of Data Mining and Knowledge Discovery
 We consider the problem of finding statistically unusual subgroups  in a multi-relation database, and extend previous work on singlerelation  subgroup discovery. We give a precise definition of the multirelation  subgroup discovery task, propose a specific form of declarative  bias based on foreign links as a means of specifying the hypothesis space,  and show how propositional evaluation functions can be adapted to the  multi-relation setting. We then describe an algorithm for this problem  setting that uses optimistic estimate and minimal support pruning, an  optimal refinement operator and sampling to ensure efficiency and can  easily be parallelized.  
Principles of Data Mining and Knowledge Discovery
 We consider the problem of finding statistically unusual subgroups  in a multi-relation database, and extend previous work on singlerelation  subgroup discovery. We give a precise definition of the multirelation  subgroup discovery task, propose a specific form of declarative  bias based on foreign links as a means of specifying the hypothesis space,  and show how propositional evaluation functions can be adapted to the  multi-relation setting. We then describe an algorithm for this problem  setting that uses optimistic estimate and minimal support pruning, an  optimal refinement operator and sampling to ensure efficiency and can  easily be parallelized.  


The Difficulty of Reduced Error Pruning of Leveled Branching Programs.
Induction of decision trees is one of the most successful approaches to supervised machine learning. Branching programs are a generalization of decision trees and, by the boosting analysis, exponentially more efficiently learnable than decision trees. In experiments this advantage has not been seen to materialize. Decision trees are easy to simplify using pruning. For branching programs no such algorithms are known. We prove that reduced error pruning of branching programs is infeasible. Finding the optimal pruning of a branching program with respect to a set of pruning examples that is separate from the set of training examples is NP-complete. Therefore, we are forced to consider approximate solutions to this problem. We also prove that finding an approximate solution of arbitrary accuracy is computationally intractable. In particular, the reduced error pruning of branching programs is APX-hard.
Experiments with Projection Learning.
This paper explores collaboration and learning between stakeholders in customer-driven student projects. The research objectives are to obtain empirically based knowledge on how students relate to stakeholders in customer-driven projects, and to suggest implications for the pedagogical design of the project courses. Empirical data was collected from two Bachelor courses in software engineering at two learning institutions in Norway. To make sense of the interaction between the three stakeholders in the project: the student groups, the university and the customer, we build on Wenger’s concept of communities of practice and on the concept of boundary objects. Our analysis highlights that students, through practical experience in the projects, learn to balance the requirements and expectations from different stakeholders in designing a working technical solution- a valuable skill for software engineers. We argue that for students to learn to balance stakeholders ’ interests in the best possible way, visibility of stakeholders’ goals should be focused throughout the projects. Explicit reference to the goals should be incorporated into project artifacts serving as boundary objects. Collaboration technologies providing standard shared workspace functionality are seen as adequate to support this. 1.
Fast Minimum Training Error Discretization.
The need to discretize a numerical range  into class coherent intervals is a problem  frequently encountered. Training Set Error  (TSE) is one of the commonly used impurity  functions in this task.
Partition%2DRefining Algorithms for Learning Finite State Automata.

Linear%2DTime Preprocessing in Optimal Numerical Range Partitioning.

On the Practice of Branching Program Boosting.
Branching programs are a generalization of decision trees. From the viewpoint of boosting theory the former appear to be exponentially more efficient. However, earlier experience demonstrates that such results do not necessarily translate to practical success. In this paper we develop a practical version of Mansour and McAllester&#039;s [13] algorithm for branching program boosting. We test the algorithm empirically with real-world and synthetic data. Branching programs attain the same prediction accuracy level as C4.5. Contrary to the implications of the boosting analysis, they are not significantly smaller than the corresponding decision trees. This further corroborates the earlier observations on the way in which boosting analyses bear practical significance.
Preprocessing Opportunities in Optimal Numerical Range Partitioning.

Robot Landmark Learning with SVMs.

On the Computational Complexity of Optimal Multisplitting.
The need to partition or discretize numeric value ranges arises in machine learning and data mining algorithms. This subtask is a potential time-consumption bottleneck, since the number of candidate partitions is exponential in the number of possible cut points in the range. Thus, many heuristic algorithms have been proposed for this task. Recently, the efficiency...
An Analysis of Reduced Error Pruning.
Top-down induction of decision trees has been observed to suffer from the inadequate functioning of the pruning phase. In particular, it is known that the size of the resulting tree grows linearly with the sample size, even though the accuracy of the tree does not improve. Reduced Error Pruning is an algorithm that has been used as a representative technique in attempts to explain the problems of decision tree learning. In this paper we present analyses of Reduced Error Pruning in three different settings. First we study the basic algorithmic properties of the method, properties that hold independent of the input decision tree and pruning examples. Then we examine a situation that intuitively should lead to the subtree under consideration to be replaced by a leaf node, one in which the class label and attribute values of the pruning examples are independent of each other. This analysis is conducted under two different assumptions. The general analysis shows that the pruning probability of a node fitting pure noise is bounded by a function that decreases exponentially as the size of the tree grows. In a specific analysis we assume that the examples are distributed uniformly to the tree. This assumption lets us approximate the number of subtrees that are pruned because they do not receive any pruning examples. This paper clarifies the different variants of the Reduced Error Pruning algorithm, brings new insight to its algorithmic properties, analyses the algorithm with less imposed assumptions than before, and includes the previously overlooked empty subtrees to the analysis.
Generalizing Boundary Points.
The complexity of numerical domain partitioning depends on the number of potential cut points. In multiway partitioning this dependency is often quadratic, even exponential. Therefore, reducing the number of candidate cut points is important. For a large family of attribute evaluation functions only boundary points need to be considered as candidates. We prove that an even more general property holds for many commonly-used functions. Their optima are located on the borders of example segments in which the relative class frequency distribution is static. These borders are a subset of boundary points. Thus, even less cut points need to be examined for these functions. The results shed a new light on the splitting properties of common attribute evaluation functions and they have practical value as well. The functions that are examined also include non-convex ones. Hence, the property introduced is not just another consequence of the convexity of a function.
Generalizing Boundary Points.
The complexity of numerical domain partitioning depends on the number of potential cut points. In multiway partitioning this dependency is often quadratic, even exponential. Therefore, reducing the number of candidate cut points is important. For a large family of attribute evaluation functions only boundary points need to be considered as candidates. We prove that an even more general property holds for many commonly-used functions. Their optima are located on the borders of example segments in which the relative class frequency distribution is static. These borders are a subset of boundary points. Thus, even less cut points need to be examined for these functions. The results shed a new light on the splitting properties of common attribute evaluation functions and they have practical value as well. The functions that are examined also include non-convex ones. Hence, the property introduced is not just another consequence of the convexity of a function.
On the Complexity of Optimal Multisplitting.
The need to partition or discretize numeric value ranges arises in machine learning and data mining algorithms. This subtask is a potential time-consumption bottleneck, since the number of candidate partitions is exponential in the number of possible cut points in the range. Thus, many heuristic algorithms have been proposed for this task. Recently, the efficiency...
On the Complexity of Optimal Multisplitting.
The need to partition or discretize numeric value ranges arises in machine learning and data mining algorithms. This subtask is a potential time-consumption bottleneck, since the number of candidate partitions is exponential in the number of possible cut points in the range. Thus, many heuristic algorithms have been proposed for this task. Recently, the efficiency...
The Biases of Decision Tree Pruning Strategies.
Post pruning of decision trees has been a successful approach in many real-world experiments, but over all possible concepts it does not bring any inherent improvement to an algorithm&#039;s performance. This work explores how a PAC-proven decision tree learning algorithm fares in comparison with two variants of the normal top-down induction of decision trees. The algorithm does not prune its hypothesis per se, but it can be understood to do pre-pruning of the evolving tree. We study a backtracking search algorithm, called Rank, for learning rank-minimal decision trees. Our experiments follow closely those performed by Schaffer [20]. They confirm the main findings of Schaffer: in learning concepts with simple description pruning works, for concepts with a complex description and when all concepts are equally likely pruning is injurious, rather than beneficial, to the average performance of the greedy topdown induction of decision trees. Pre-pruning, as a gentler technique, settles in the ...
Predicting the Speed of Beer Fermentation in Laboratory and Industrial Scale.
. Characteristic of the beer production process is the uncertainty  caused by the complex biological raw materials and the yeast, a  living organism. This uncertainty is exemplied by the fact that predicting  the speed of the beer fermentation process is a non-trivial task.  We employ neural network and decision tree learning to predict the speed  of the beer fermentation process. We use two data sets: one that comes  from laboratory-scale experiments and another that has been collected  from an industrial scale brewing process. In the laboratory-scale experiments  a neural network that employs characteristics of the ingredients  and the condition of the yeast, could predict the fermentation speed  within 2% of the true value. Decision trees for classifying whether the  speed of fermentation will be slow or fast were constructed from the same  data. Astonishing simple decision trees were able to predict the classes  with 95%98% accuracy. In contrast to the neural net experiment, even  ...
Speeding Up the Search for Optimal Partitions.
. Numerical value range partitioning is an inherent part of inductive learning. In classification problems, a common partition ranking method is to use an attribute evaluation function to assign a goodness score to each candidate. Optimal cut point selection constitutes a potential efficiency bottleneck, which is often circumvented by using heuristic methods. This paper aims at improving the efficiency of optimal multisplitting. We analyze convex and cumulative evaluation functions, which account for the majority of commonly used goodness criteria. We derive an analytical bound, which lets us filter out---when searching for the optimal multisplit---all partitions containing a specific subpartition as their prefix. Thus, the search space of the algorithm can be restricted without losing optimality. We compare the partition candidate pruning algorithm with the best existing optimization algorithms for multisplitting. For it the numbers of evaluated partition candidates are, on the averag...
General and Efficient Multisplitting of Numerical Attributes.
. Often in supervised learning numerical attributes require special treatment and do not fit the learning scheme as well as one could hope. Nevertheless, they are common in practical tasks and, therefore, need to be taken into account. We characterize the well-behavedness of an evaluation function, a property that guarantees the optimal multi-partition of an arbitrary numerical domain to be defined on boundary points. Well-behavedness reduces the number of candidate cut points that need to be examined in multisplitting numerical attributes. Many commonly used attribute evaluation functions possess this property; we demonstrate that the cumulative functions Information Gain and Training Set Error as well as the non-cumulative functions Gain Ratio and Normalized Distance Measure are all well-behaved. We also devise a method of finding optimal multisplits efficiently by examining the minimum number of boundary point combinations that is required to produce partitions which are optimal wit...
Postponing the Evaluation of Attributes with a High Number of Boundary Points.
. The efficiency of the otherwise expedient decision tree learning  can be impaired in processing data-mining-sized data if superlineartime  processing is required in attribute selection. An example of such a  technique is optimal multisplitting of numerical attributes. Its efficiency  is hit hard even by a single troublesome attribute in the domain.  Analysis shows that there is a direct connection between the ratio of the  numbers of boundary points and training examples and the maximum  goodness score of a numerical attribute. Class distribution information  from preprocessing can be applied to obtain tighter bounds for an attribute  &#039;s relevance in class prediction. These analytical bounds, however,  are too loose for practical purposes.  We experiment with heuristic methods which postpone the evaluation of  attributes that have a high number of boundary points. The results show  that substantial time savings can be obtained in the most critical data  sets without having to give up...
A Review of How the Mind Works.

Well%2DBehaved Evaluation Functions for Numerical Attributes.

Efficient Multisplitting on Numerical Data.

On the Well%2DBehavedness of Important Attribute Evaluation Functions.

TELA A Platform for Experimenting with Attribute%2Dbased Learning Programs.

A Geometric Approach to Feature Selection.

In Defense of C4.5 Notes Learning One%2DLevel Decision Trees.
We discuss the implications of Holte&#039;s recentlypublished article, which demonstrated that on the most commonly used data very simple classification rules are almost as accurate as decision trees produced by Quinlan&#039;s C4.5. We consider, in particular, what is the significance of Holte&#039;s results for the future of top-down induction of decision trees. To an extent, Holte questioned the sense of further research on multilevel decision tree learning. We go in detail through all the parts of Holte&#039;s study. We try to put the results into perspective. We argue that the (in absolute terms) small difference in accuracy between 1R and C4.5 that was witnessed by Holte is still significant. We claim that C4.5 possesses additional accuracy-related advantages over 1R. In addition we discuss the representativeness of the databases used by Holte. We compare empirically the optimal accuracies of multilevel and one-level decision trees and observe some significant differences. We point out several defici...
